Derived experiment_design.csv and cross-version parity suite (Stage 1, PR 10) - #196

Stacked on #195 (the re-pin to Spec 7dbae0c, which brings the golden files this PR compares against). The base is that branch so the diff shows only this PR. Retarget to main after #195 merges.

What

Stage 1 design PR 10 (design sections 3 "Derived experiment_design.csv", 4 "Compatibility", row 10; conformance P30 and P32, derived part). Additive: no run, notebook or CLI command uses it yet.

• New module design_derive (layer io): derive_design(description) -> ExperimentDesign and write_derived_design(description, path), per Spec section 11:
  • condition is the condition's source_folder, biological_repeat the member's source_repeat, repeat_group the repeat ID;
  • control is true for a condition that another condition names as its reference;
  • image_count is the member's, technical_repeat_count is 0 for the single unit single and else the number of units;
  • confirmed is true (the reader refuses a published repeat that is not) and confirmed_on is the repeat's date.
• Delimiter, quoting, final LF and no BOM are today's, because it is written by the existing write_experiment_design; the sort is its own (condition, then repeat, code point order). The file carries folder labels and no condition, technical-repeat or channel ID, so renaming such an ID never changes it. Nothing is guessed, defaulted or repaired.
• tests/test_spec_derived_design.py (77 tests): the derived file equals the Spec golden files for v13, v33 and v34 byte for byte, and equals the reference validator's derive_csv on every valid fixture (36); every derived file is read back by read_confirmed_design.
• tests/test_parity_v0_4_0.py (24 tests) with tests/parity_datasets.py: the same confirm-run-rerun script runs under the v0.4.0 tag (tests/released_code.py, which fails rather than skips in CI) and under this checkout, on six legacy homogeneous datasets: one repeat, two conditions by two repeats, technical repeats, flat, OME names, RGB. For each:
  • plane_histograms.csv and experiment_design.csv are byte-identical (so plane IDs, FOV keys and position_id are the same);
  • every manifest file entry keeps every v0.4.0 key and value; the only key added since is the optional raw_channel_names (asserted as the allowed difference);
  • the rerun in the same folder reads no file, and this code over a copy of a v0.4.0 output folder recomputes zero rows (nothing new, changed, resumed or read, no rebuild reason, table bytes unchanged);
  • the derived design of the dataset's description (IDs equal to the folder labels, as in v13) equals the bytes v0.4.0 wrote.

Notes for review

• The two known specified deltas of design section 4 (an unreferenced control exports control = false; a group with several legacy dates takes the latest) are not exercised: every dataset above has one control referenced by all other conditions, or none.
• Manifest version differs by design (the strict default writes version 3), so manifests are compared per file entry, not byte for byte; the CSVs are byte for byte.
• A flat experiment writes a one-row design (single, single) in both versions when a design is passed to the run.
• design.py is untouched (it is already over the line limit); the new code lives in the new module.

Tests

On the base branch the two new test files fail at collection (ModuleNotFoundError: morphobricks_preprocessing.design_derive). Here: 77 and 24 passed.

python tools/check.py: 4041 passed, 4 skipped, code rules clean; ruff check and ruff format --check clean. The 4 skips are environment ones (case-insensitive file system and file-mode tests, unrelated to this change).

Generated by Claude Code